Methods, apparatus and systems for monitoring devices

ABSTRACT

Methods, apparatus, systems and articles of manufacture are disclosed herein including a monitoring system an image sensor to obtain image data of a device and a governor to cause the image sensor to obtain image data of the device, to form an impression from the image data, to use the impression and the image data to determine a verdict.

FIELD OF THE DISCLOSURE

This disclosure relates generally to monitoring, and, more particularly,to methods, apparatus and systems for monitoring devices.

BACKGROUND

Conventional monitoring systems typically rely on alerts from sensorsand analysis of event streams to infer that an anomalous condition hasmanifested. However, mechanical malfunctions may present an indirect anddiffused correlation on behavioral parameters over a long period oftime. Consequently, mechanical malfunctions may escape detection untilsuch time as a serious degradation exists.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of an example device for monitoringsystems and structures implemented in an example vehicle constructed inaccordance with some teachings of this disclosure.

FIG. 2 is a block diagram of an example implementation of the governor150 of FIG. 1.

FIGS. 3A-3C illustrate example implementations of example image sensorsto provide example image data to the example governor of FIG. 1 inaccordance with some teachings of this disclosure.

FIGS. 4A-4B illustrate example image data from the example image sensorsof FIGS. 3A-3C at a first time and at a second time, respectively, inaccordance with some teachings of this disclosure.

FIGS. 5A-5B present flowchart representations of computer-executableinstructions that may be executed to implement the example governor ofFIGS. 1-2.

FIG. 6 illustrates a representation of implementation of the exampleimage data from the example image sensors of FIGS. 1 and/or 3A-3C by theexample governor of FIGS. 1-2 in accord with the example instructions ofFIGS. 5A-5B.

FIG. 7 illustrates an example training model used to form an exampleimpression from the example image data from the example image sensors ofFIGS. 1 and/or 3A-3C by the example governor of FIGS. 1-2.

FIG. 8 illustrates an example implementation of the example image datafrom the example image sensors of FIGS. 1 and/or 3A-3C by the examplegovernor of FIGS. 1-2 in accord with the example instructions of FIGS.5A-5B.

FIG. 9 is a block diagram illustrating an example processor platformwhich may execute the instructions of FIGS. 5A-5B to implement theexample governor of FIGS. 1-2.

The figures are not to scale. As used in this patent, stating that anypart (e.g., a layer, film, area, or plate) is in any way positioned on(e.g., positioned on, located on, disposed on, or formed on, etc.)another part, indicates that the referenced part is either in contactwith the other part, or that the referenced part is above the other partwith one or more intermediate part(s) located therebetween. Stating thatany part is in contact with another part means that there is nointermediate part between the two parts.

DETAILED DESCRIPTION

Conventional monitoring systems may fail to detect degradation of amechanical system. In manned systems, such as a vehicle with a driver,vehicle sensory data is supplemented by humans who may notice subtlealterations in vehicle performance, unusual noises, unusual smells, orthe like, and who may assess the need to investigate further. Suchconventional monitoring systems use data streams from system sensors(e.g., engine rotation speed, vibration, tire pressure, etc.) asindicators, but these data streams at times prove insufficient todiscover a problem or indicate a severity of a problem. Autonomoussystems likewise receive data streams from internal sensors and analyzethe data streams to draw inferences therefrom. This monitoring paradigmand analysis is suitable for simple mechanisms of degradation. Forinstance, a low tire pressure indicator is sufficient to alert as to alow tire pressure. However, a variety of mechanical failure mechanismscan prove difficult to diagnose via on-board sensors. For instance, ablown head gasket could manifest in a variety of ways (e.g., misfires,lowered compression, overheating, swelling of the radiator cap,corruption of fluids, oil leak, coolant leak, whitish exhaust, etc.).

In accord with the some teachings of this disclosure, autonomous devices(e.g., autonomous land vehicles, autonomous aerial vehicles, drones,robots, spacecraft, industrial equipment or machinery, etc.) and/ormanned devices (e.g., terrestrial vehicles, aircraft, watercraft, etc.)implement an example governor to monitor one or more systems,subsystems, or components to assess damage and/or degradation of theapparatus/device, including any system(s), subsystem(s), orcomponent(s). The example governor helps to eliminate the humanin-the-loop performing manual screening of autonomous devices and helpsto increase device autonomy.

FIG. 1 is a schematic illustration of an example system 100 formonitoring systems and structures implemented in an example device 110,which may be autonomous or manned. While the example of FIG. 1 depicts adevice 110, the teachings herein likewise apply to other types ofautonomous devices and/or manned devices, such as those noted above.

As shown in FIG. 1, the device 110 includes one or more example imagesensors 120 (hereinafter “image sensor 120” for brevity) in one or moreareas of the vehicle (e.g., one or more areas of an engine compartment,motor(s), undercarriage, brake system, etc.). In some examples, theimage sensor 120 includes a thermal image sensor, a spatial image sensorand/or an optical image sensor. FIG. 1 also shows the device 110 toinclude one or more example sensors 125 (e.g., one or more pressuresensor(s), one or more vibration sensor(s), one or more velocitysensor(s) and/or one or more acceleration sensor(s), etc.) in one ormore areas of the vehicle to provide telemetry data for one or moresystems or subsystems of the device 110.

The image sensor 120 is to obtain example image data, such as thermalimage data, spatial image data and/or optical image data, of the area(s)of the device 110 in which the image sensor 120 is disposed. In someexamples, the image sensor 120 may include the non-contact MLX90620temperature measurement device from Melexis of Belgium, which includes a16×4 element far infrared (FIR) thermopile sensor array constructed toproduce a real-time map of heat values. In some examples, the imagesensor 120 includes a spatial image sensor like Intel® RealSense™ DepthModule D400.

The image sensor 120 outputs the image data via an example communicationpathway 130, such as a hardwired communication pathway or a wirelesscommunication pathway, to an example governor 150. In some examples, thegovernor 150 is disposed within the device 110. For instance, thegovernor 150 may be disposed in a dashboard, under a seat, or in a trunkof the device 110. In some examples, the governor 150 is disposed at aremote location (e.g., external to the device 110, in a different regionthan the device 110, etc.). As described below, the governor 150processes the image data from the image sensor 120 and outputs the imagedata and/or a derivative thereof, via a communication device 155, to anexample RF broadcast tower 160 and/or an example network 165.

In some examples, the communication device 155 includes a device such asa transmitter, a transceiver, a modem and/or network interface card tofacilitate exchange of the image data with one or more external machines170 (e.g., computing devices of any kind, computer, server, etc.) viathe RF broadcast tower 160 and/or network 165. In some examples, thecommunication device 155 may communicate, directly or indirectly (e.g.,via one or more intermediary devices), to the network 165 via anEthernet connection, a digital subscriber line (DSL), a telephone line,coaxial cable, a cellular telephone system, a 10Base-T connection, aFireWire connector or a Universal Serial Bus (USB) connector. Thus,while an example RF broadcast tower 160 and an example communicationdevice 155 are indicated in the example of FIG. 1, in some examples theexample governor 150 is connected to one or more external machines 170via a hardwired connection (e.g., a USB connection).

FIG. 2 is a block diagram of an example implementation of the governor150 of FIG. 1. In the example implementation of FIG. 2, the governor 150includes an example image manager 210, an example impression manager 220and an example impression comparator 230.

In general, the example governor 150 is to cause the image sensor 120 toobtain example image data of the structure of the device 110 at a firsttime, from which an example impression may be formed. The impressionfacilitates comparison of actual data from sensor data with datacalculated as a result of applying a trained model or impression forprevious data. For instance, in a first example, where previous samplesof data at times t-1, t-2, . . . t-N are known and the example governor150 receives sensor data at time t, the governor 150 applies theimpression to the data samples at moments times t-1, t-2, . . . t-N andcalculates estimated data for time t using the impression. The governor150 then compares the estimated data at time t with the actual data attime t and determines a verdict as to whether the comparison isfavorable (e.g., a “good” state) or unfavorable (e.g., a “bad” state).In a second example, where previous samples of data at times t-1, t-2, .. . t-N are known and the example governor 150 receives sensor data attime t, the governor 150 applies the impression to the data samples atmoments times t, t-1, t-2, . . . t-N and determines a verdict as towhether the comparison is favorable (e.g., a “good” state), unfavorable(e.g., a “bad” state) or “unknown.” The impression can be updatedcontinuously after every sample of data, periodically, or aperiodically.For instance, in some examples, the governor 150 updates the impressionperiodically using batches of data. In some examples, the governor 150updates the impression when a new classification becomes available inresponse to telemetry data or an external expert system. The impressionallows the governor 150 to determine if measured data is close to datathat is predicted, with the governor 150 calculating a verdict (e.g.,“good” state or “bad” state) through a comparison of predicted data andmeasured data (e.g., a neural net “feedforward” evaluation) andmodifying or updating the impression to integrate input data of whichthe governor 150 was not previously familiar (e.g., a neural net“backpropagation” or training).

The example image manager 210 is to receive and process example imagedata from the image sensor 120 and to pass the image data to the exampleimpression manager 220 for processing. In some examples, the exampleimage manager 210 obtains image data of the structure of the device 110responsive to a request from the governor 150.

The example impression manager 220 is to use the image data to form animpression or trained model of the structure of the device 110 imaged bythe image sensor 120. In some examples, each volume and/or surface areain a field of view, or points of view, of the image sensor 120 isassigned a value corresponding to the image data obtained by the imagesensor 120. In some examples, the impression can be a set of weights ina matrix representing a neural network (e.g., Artificial Neural Network(ANN), Recurrent Neural Network (RNN), Convolutional Neural Network(CNN), Deep Neural Network (DNN), etc.) or some other representation ofbehavior (e.g., a “decision tree,” Support Vector Machine (SVM),Logistic Regression (LR), etc.). The impression is trained (e.g., matrixcoefficients are updated, etc.) using data from the image sensor 120.For example, a thermal image sensor would output image data including atemperature reading at each point or pixel in a field of view (e.g., a640×480 thermal image sensor may include 307,200 pixels) and a spatialimage sensor would output distances (e.g., via time-of-flight) at eachpoint or pixel in a field of view (e.g., a point cloud of a 3D imagesensor, etc.). The point cloud data for temperature and/or distance isbrought into a common reference system (e.g., polar, Cartesian, etc.).The result of training is a modified impression which accommodates orintegrates valid changes in image data.

In some examples, an impression may include an arrangement of raw data,such as a temperature in a volume or surface area defined by a selectedcoordinate system, or a derivative of the raw data, such as a map ofspatial thermal gradients within a given volume and/or a mapping of thespatial thermal gradients onto a coefficients vector (e.g., a Radontransform). The example impression manager 220 may use any manner ofexpression of the image data to uniquely identify the imaged operationalstate. For instance, the impression manager 220 may convert the imagedata into an alternative representation via a mathematical transform,linear transform, matrix representation, linear mapping, eigenvaluedecomposition, wavelet decomposition, geometric multiscale analysis,polygonal 3D model, surface model, non-uniform rational basis spline(NURBS) surface model, polygon mesh, and store and/or export therepresentation in a suitable format (e.g., as a Standard TessellationLanguage (STL) file, Standard ACIS Text (SAT) file and/or OBJ geometryfile, or any other 3D modelling file format).

In some examples an initial impression is provided by a manufacturer ofthe device 110 (e.g., a vehicle, etc.) and the initial impression isupdated using image data from the image sensor 120.

In some examples, the example impression comparator 230 is to apply animpression to samples of image data at times t-1, t-2, . . . t-N and tocalculate estimated image data for time t. The impression comparator 230then compares the estimated image data for time t to actual image datafrom the imager 120 for time t to determine a level of correspondencebetween the estimated image data for time t to the actual image data attime t. The impression comparator 230 then renders a verdict as towhether or not the actual data corresponds to a “known good” state or a“known bad” state. In some examples, the example impression comparator230 is to apply an impression to samples of image data at times t, t-1,t-2, . . . t-N and is to calculate a verdict estimated image data fortime t (e.g., a “known good” state, a “known bad” state, an “unknown”state, etc.).

In some instance, the impression manager 220 maps spatial gradientsand/or thermal gradients onto a coefficients vector (e.g., by a Radontransform, Hough transform, Funk transform, combinations of transforms,etc.) to form and/or update the impression and the impression comparator230 is to compare basis vectors in a vector space as between sets ofimage data at different times to determine to detect correspondencebetween the image data and a known state (e.g., good state, a bad state,etc.) and/or an unknown state.

In some examples, the governor 150 outputs an example impression and/orthe image data relating to the impression, or derivatives thereof, to anexample memory 250. The example memory 250 includes, in an exampleoperating state manager 252 corresponding to an operating state of thedevice, the example impression 254, including image data relating to theimpression 254 and/or derivatives thereof. In some examples, theoperating state manager 252 differentiates between a plurality ofoperating states (e.g., one or more “known good” state(s), one or more“known bad” state(s), etc.). The operating state manager 252 alsoincludes an example first image data set 256 and successive image datasets to the first image data set 256 through an example N^(th) imagedata set 258, where N is any integer. In some examples, the first imagedata set 256 and/or another other image data set through the N^(th)image data set 258 include more than one image data set (e.g., aplurality of image data sets from a plurality of different times). Insome examples, the first image data set 256 is preloaded into the memory250 by a vendor of the device 110.

In some examples, the memory 250 is local to the device 110. In someexamples, the memory 250 is remote to the device 110 and communicationbetween the governor 150 and the memory 250 is via the communicationdevice 155 and/or via the communication device 155 and any intermediarydevices such as the RF broadcast tower 160 and/or the network 165.

While an example manner of implementing the governor 150 of FIG. 1 isillustrated in FIG. 2, one or more of the elements, processes and/ordevices illustrated in FIG. 2 may be combined, divided, re-arranged,omitted, eliminated and/or implemented in any other way. Further, theexample image manager 210, the example impression manager 220 and/or theexample impression comparator 230 and/or, more generally, the examplegovernor 150 of FIG. 2 may be implemented by hardware, software,firmware and/or any combination of hardware, software and/or firmware.Thus, for example, any of the example image manager 210, the exampleimpression manager 220 and/or the example impression comparator 230and/or, more generally, the example governor 150 could be implemented byone or more analog or digital circuit(s), logic circuits, programmableprocessor(s), application specific integrated circuit(s) (ASIC(s)),programmable logic device(s) (PLD(s)) and/or field programmable logicdevice(s) (FPLD(s)). When reading any of the apparatus or system claimsof this patent to cover a purely software and/or firmwareimplementation, at least one of the example image manager 210, theexample impression manager 220 and/or the example impression comparator230 and/or, more generally, the example governor 150 is/are herebyexpressly defined to include a non-transitory computer readable storagedevice or storage disk such as a memory, a digital versatile disk (DVD),a compact disk (CD), a Blu-ray disk, etc. including the software and/orfirmware. Further still, the example governor 150 of FIG. 1 may includeone or more elements, processes and/or devices in addition to, orinstead of, those illustrated in FIG. 2, and/or may include more thanone of any or all of the illustrated elements, processes and devices.

A flowchart representative of example machine readable instructions forimplementing the governor 150 of FIG. 2 is shown in FIGS. 5A-5B. In thisexample, the machine readable instructions comprise a program forexecution by a processor such as the processor 912 shown in the exampleprocessor platform 900 discussed below in connection with FIG. 9. Theprogram may be embodied in software stored on a non-transitory computerreadable storage medium such as a CD-ROM, a floppy disk, a hard drive, adigital versatile disk (DVD), a Blu-ray disk, or a memory associatedwith the processor 912, but the entire program and/or parts thereofcould alternatively be executed by a device other than the processor 912and/or embodied in firmware or dedicated hardware. Further, although theexample program is described with reference to the flowchart illustratedin FIGS. 5A-5B, many other methods of implementing the example governor150 may alternatively be used. For example, the order of execution ofthe blocks may be changed, and/or some of the blocks described may bechanged, eliminated, or combined. Additionally or alternatively, any orall of the blocks may be implemented by one or more hardware circuits(e.g., discrete and/or integrated analog and/or digital circuitry, aField Programmable Gate Array (FPGA), an Application Specific Integratedcircuit (ASIC), a comparator, an operational-amplifier (op-amp), a logiccircuit, etc.) structured to perform the corresponding operation withoutexecuting software or firmware.

As mentioned above, the example processes of FIGS. 5A-5B may beimplemented using coded instructions (e.g., computer and/or machinereadable instructions) stored on a non-transitory computer and/ormachine readable medium such as a hard disk drive, a flash memory, aread-only memory, a compact disk, a digital versatile disk, a cache, arandom-access memory and/or any other storage device or storage disk inwhich information is stored for any duration (e.g., for extended timeperiods, permanently, for brief instances, for temporarily buffering,and/or for caching of the information). As used herein, the termnon-transitory computer readable medium is expressly defined to includeany type of computer readable storage device and/or storage disk and toexclude propagating signals and to exclude transmission media.“Including” and “comprising” (and all forms and tenses thereof) are usedherein to be open ended terms. Thus, whenever a claim lists anythingfollowing any form of “include” or “comprise” (e.g., comprises,includes, comprising, including, etc.), it is to be understood thatadditional elements, terms, etc. may be present without falling outsidethe scope of the corresponding claim. As used herein, when the phrase“at least” is used as the transition term in a preamble of a claim, itis open-ended in the same manner as the term “comprising” and“including” are open ended.

FIG. 3A shows an example image sensor 302 disposed in an example enginecompartment structure 304. The example image sensor 302 includes anexample laser 310, powered by a power source (not shown), disposed toemit an example beam of collimated light 315 at an example photosensor320. In some examples, the photosensor 320 includes a photodiode. Thelaser 310 may include any laser of any wavelength (e.g., any wavelengthbetween 650-1550 nm) and may include, for example, a diode laser or asemiconductor laser. In some examples, the image sensor 302 includes anAdafruit VL53L0X Time of Flight Distance Sensor. In some examples, thepower source for the laser 310 includes a battery of the device (e.g.,autonomous and/or manned device, device 110, etc.) implementing theimage sensor 302. The image sensor 302 and the photosensor 320 form partof an optical circuit in which photons of the incident beam ofcollimated light 315 are converted into current representative of analignment between the laser 310 and the photosensor 320. A change in anintensity of the incident beam of collimated light 315 on thephotosensor 320 may indicate a shift of one part of the example enginecompartment structure 304 relative to another part of the enginecompartment structure 304.

In some examples, the image sensor 302 includes a plurality of imagesensors 302 producing vectors of measured values, with each of the imagesensors 302 producing scalar values combinable with respect to time as atuple on n-vector, where n is any integer.

FIG. 3B shows an example image sensor 325 disposed in an example enginecompartment structure 304. In some examples, the example image sensor325 is a time-of-flight image sensor such as, but not limited to, arange gated image sensor, a direct time-of-flight image sensor, or asonic range finder. The example image sensor 325 includes an examplelaser 310, powered by a power source (not shown), disposed to emit anexample beam of collimated light 315 at an object 330. The example imagesensor 325 also includes a photosensor 320 to receive reflected lightfrom the object 330 responsive to the incident beam of collimated light315. In some examples, the photosensor 320 includes a collection lensdisposed adjacent the laser 310 to focuses the incident light theretoonto a solid-state photodiode (e.g., a linear array camera, a CMOSarray, etc.). The laser 310 may include any laser of any wavelength(e.g., any wavelength between 650-1550 nm) and may include, for example,a diode laser or a semiconductor laser. In some examples, the powersource for the laser 310 includes a battery of the device (e.g.,autonomous and/or manned device, device 110, etc.) implementing theimage sensor 302. The image sensor 302 and the photosensor 320 form partof an optical circuit in which photons of the incident light 332 areconverted into current representative of a distance between the imagesensor 325 and the object 330, which is in turn representative of analignment between the image sensor 325 and the object 330. A change inan intensity of the incident light 332 on the photosensor 320 and/orphase shift of the incident light 332 may indicate a shift of one partof the example engine compartment structure 304 relative to another partof the engine compartment structure 304. A combination of distance databetween multiple points enables development of a computational geometryof constituent parts.

FIG. 3C shows an example image sensor 340 disposed in an example enginecompartment structure 304. In some examples, the example image sensor340 is a Light Detection and Ranging (LIDAR) device or a 3D-cameraincluding an example laser scanner 345 and/or light source. In someexamples, the laser scanner 345 is to emit a scanning beam 351 ofcollimated light across a selected volume of the structure of the device(e.g., a volume within an engine compartment of a device 110, etc.) toproduce a point cloud of distance information or a depth map using phaseshifts in return signals to the laser scanner 345. For instance, FIG. 3Cillustrates, responsive to illumination by the image sensor 340, anexample first cone 360 of reflected light, an example second cone 362 ofreflected light and an example third cone 364 of reflected lightincident, respectively, from an example first object 370, an examplesecond object 372 and an example third object 374 of the structure ofthe autonomous and/or manned device.

The reflected light from the example first object 370, the examplesecond object 372 and the example third object 374 interacts with one ormore lenses and photosensors of the image sensor 340 yielding, viatime-of-flight, distance data for each point in the point cloud. Theimage sensor 340 is to build a map (e.g., a distance map and/or athermal map) of a selected volume of the structure of the autonomousand/or manned device (e.g., a volume within the engine compartment,etc.) from which the governor 150 can detect deviations in alignmentand/or temperature. In some examples, the light source 350 includes oneor more lights to illuminate a selected volume of the structure of theautonomous and/or manned device (e.g., a volume within the enginecompartment, etc.). In some examples, the light source 350 includes asolid-state diode, a lamp, and/or a bulb to output light in one or moreranges of wavelengths (e.g., visible light spectrum, infrared lightspectrum, etc.). In some examples, the image sensor 340 includesRealSense™ technology from Intel®, such as an Intel® RealSense™ DepthModule D400 Series image sensor.

In some examples, the impression manager 220 integrates or associatesthe image data from the image sensor 120 (e.g., image sensor 302, 325,340) with telemetry data from one or more additional sensors 125 of theautonomous and/or manned device. For instance, image data used by theimpression manager 220 to form and/or update the impression 254 may beassociated with data from one or more other sensors 125 (e.g., apressure sensor, a vibration sensor, a velocity sensor, an accelerationsensor, etc.) operatively associated with one or more systems orsubsystems of the device (e.g., autonomous and/or manned device, device110). The impression comparator 230 is thus informed as to changes in anoperating condition and is able to contextually determine whether arendered verdict is indicative of a changed operating condition or isindicative of a potential malfunction.

FIGS. 4A-4B illustrate example image data from one or more example imagesensors 120 distributed in one or more locations of a selected volume405 of a structure of the autonomous and/or manned device. In theexample of FIGS. 4A-4B, the selected volume 405 includes an enginecompartment of a device 110 (e.g., an engine-based vehicle) and theselected structure 410 includes an example engine. In FIGS. 4A-4B, adevice 110 hood is removed for clarity. In some examples, one or moreimage sensors 120 are disposed on an underside of the hood to face theselected volume 405 and the selected structure 410.

FIG. 4A represents example image data at a first time and FIG. 4Brepresents example image data at a second time. In FIG. 4A, a first setof components is shown to have a first temperature gradient 420, asecond set of components is shown to have a second temperature gradient430 and a third set of components, including an example component 445,is shown to have a third temperature gradient 440, the temperaturegradients being represented by different degrees of fill. In theillustrated example, the second temperature gradient 430 is greater thanthe first temperature gradient 420 and the third temperature gradient440 is greater than the second temperature gradient 430.

In FIG. 4B, the first set of components is shown to have the firsttemperature gradient 420, the second set of components is shown to havethe second temperature gradient 430 and a third set of components isshown to have the third temperature gradient 440. However, as comparedto FIG. 4A, FIG. 4B shows the component 445 in the third set ofcomponents has a fourth temperature gradient 450 higher than the thirdtemperature gradient 440.

Example comparisons utilizing the example data of FIGS. 4A-4B aredescribed, by way of example, in FIGS. 5A-5B and FIGS. 6-8.

The programs or instructions of FIGS. 5A-5B begins with program 500 atexample block 505 in FIG. 5A, where the image manager 210 of thegovernor 150 receives N-sets of image data from the image sensor 120(e.g., 302, 325, 340), where N represents any integer (e.g., 1, 2, 3,etc.). At example block 510 of FIG. 5A, the image manager 210 and/orgovernor 150 receives N-sets of telemetry data from the sensor(s) 125.In some examples, the image manager 210 and/or governor 150 may performpre-processing of the image data from the image sensor 120 and/ortelemetry data from the sensors 125 at block 515. The pre-processing maybe used, for example, to suppress distortions in the data, eliminatenoisiness in the data, enhance the data and/or normalize the data. Forinstance, the image manager 210 may transform the image data into anappropriate coordinate system and into a format that is suitable forfurther processing by the image manager 210 and/or the impressionmanager 220 (e.g., converting pointwise laser scanner data into rastermodels or other format acceptable to downstream processing, correctinggrey level values of image data, executing edge detection andsegmentation methods to recognize homogeneous regions, employingclassification methods to classify regions of the structure(s)represented by the image data, etc.).

The impression manager 220 is then used to form an impression 254 orupdate the impression 254 at example block 520 of FIG. 5A and to storethe impression 254 at example block 522 in a physical, non-transientstorage medium. In some examples, the impression manager 220 receivesimage data from the image sensor 120 of FIG. 1 and/or image sensors 302,325 and/or 340 of FIGS. 3A-3C and converts the image data, at block 520,into an impression 254 including an array of data vectors representativeof the image data for the selected volume 405 and/or selected structure410 (e.g., a volume in space in the engine compartment of device 110,etc.). In some examples, following the forming of the impression 254 orupdating of the impression 254 at block 520 of FIG. 5A, control passesback to block 505 for receipt of additional image data.

FIG. 5B shows example instructions 524 beginning with example block 525,where the image manager 210 of the governor 150 receives image data(e.g., one set of image data, N-sets of image data, etc.) from the imagesensor 120 (e.g., 302, 325, 340). At block 530, the image manager 210and/or the governor 150 receives telemetry data (e.g., one set of imagedata, N-sets of image data, etc.) from the sensor(s) 125.

In some examples, at example block 535 and/or example block 540, theimage manager 210 and/or governor 150 may perform pre-processing of theimage data from the image sensor 120 and/or telemetry data from thesensors 125, respectively. The pre-processing may be used, for example,to suppress distortions in the data, eliminate noisiness in the data,enhance the data, normalize the data and/or transform the image datainto an appropriate coordinate system and into a format suitable forfurther processing, such as by the impression manager 220 and/or theimpression comparator 230.

At example block 545, the impression comparator 230 applies theimpression 254 to the image data. For instance, the governor 150 may usethe impression manager 220 to access the impression 254 from theoperating state manager 252 and apply the impression 254 to previoussamples of image data at times t-1, t-2, . . . t-N. As noted above, theimpression 254 may include, for example, a mapping of spatial gradientsand/or thermal gradients onto a coefficients vector or a set of weightsin a matrix representing a neural network (e.g., ANN/RNN/CNN/DNN,“decision tree,” SVM, LR, etc.). At example block 545, the impressioncomparator 230 may also apply the impression 254 to calculate estimateddata for time t based on the impressions applied to the previous samplesof image data at times t-1, t-2, . . . t-N. At block 550, the impressioncomparator 230 determines a verdict using the impression and the imagedata, such as by comparing the impression 254 of the estimated data attime t with the actual data at time t. At block 555, the impressioncomparator 230 renders a verdict as to whether the comparison of theimpression 254 of the estimated data at time t with the actual data attime t is favorable and reflects a good verdict (e.g., reflective of aknown “good” state of the device 110). In some examples, where previoussamples of data at times t-1, t-2, . . . t-N are known, the impressioncomparator 230 applies the impression 254 to the data samples at momentstimes t, t-1, t-2, . . . t-N at block 550 to determine a verdict anddetermines at block 555 whether the verdict corresponds to a goodverdict or a bad verdict.

If the result at block 555 is “YES,” the control passes to example block560, where the governor 150 and/or the impression comparator 230determines whether the telemetry data from the sensor(s) 125 of thedevice 110 is within acceptable limits. If the result at block 555 is“NO,” the control passes to example block 565, where the impressioncomparator 230 renders a verdict as to whether the comparison of theimpression 254 of the estimated data at time t with the actual data attime t is unfavorable and reflects a bad verdict (e.g., reflective of aknown “bad” state of the device 110). In some examples, where previoussamples of data at times t-1, t-2, . . . t-N are known, the impressioncomparator 230 applies the impression 254 to the data samples at momentstimes t, t-1, t-2, . . . t-N at block 550 and determines a verdict atblock 555 as to whether the comparison is unfavorable and reflects a badverdict.

If the result at block 565 is “NO,” control passes to example block 570,where the impression comparator 230 and/or governor 150 stores the imagedata in the memory 250 for later evaluation. Control then passes toexample block 560, discussed above, where the governor 150 and/or theimpression comparator 230 determine whether the telemetry data from thesensor(s) 125 of the device 110 is within acceptable limits.

At block 560, if the telemetry data from the sensor(s) 125 of the device110 is within acceptable limits (i.e., the result is “YES”), controlpasses to example block 575, where the impression comparator 230 and/orthe governor 150 updates the impression 254 to incorporate the imagedata as “good” (e.g., reflecting a normal operational condition,reflecting a normal structural condition, etc.). Control then passes toblock 525. If the telemetry data from the sensor(s) 125 of the device110 is not within acceptable limits and the result at block 560 is “NO,”control passes to example block 580. At block 580, the impressioncomparator 230 and/or the governor 150 updates the impression 254 toincorporate the image data as “bad” (e.g., reflecting an abnormaloperational condition, reflecting an abnormal structural condition,etc.). Control then passes back to block 585. At block 585, theimpression comparator 230 and/or governor 150 stores the image data inthe memory 250 for later evaluation. Control then passes to exampleblock 590. At block 590 the governor 150 and/or the impressioncomparator 230 output a deviation report relating to the impression 254and/or the image data (e.g., reporting locally to a controller,reporting remotely to a central facility or server, etc.). If aparticular deviation report (e.g., a temperature in a particularlocation of a structure or system of the autonomous or manned device, adisplacement of a structure or system of the autonomous or manneddevice, etc.) is later positively associated with a particularperformance issue and/or maintenance issue, the impressions and/or imagedata can then be flagged as a known problem impression. Particularly fora plurality of similarly configured devices 110 (e.g., a fleet ofdrones, a fleet of vehicles, etc.), a deviation report issued by onedevice 110 informing of a potential and/or actual performance and/ormaintenance issue may enable trending of problems, targeted preventivemaintenance, and timely corrective actions not only for the one device110, but also for the other similarly configured devices 110.

FIG. 6 illustrates a representation of example image data 600 from theimage sensor 120 of FIGS. 1 and/or 3A-3C by the governor 150 of FIGS.1-2 in accord with the example flowchart of FIGS. 5A-5B. In FIG. 6, aseries of input data vectors 610A-610F, C_(L)(t₀)-C_(L)(t₅), representimage data output by image sensor 120 at five different times (i.e.,t₀-t₅) over a selected volume 405 (e.g., a 4×4 volume in space in theengine compartment of device 110, etc.). In some example, the input datavectors 610A-610N, C_(L)(t₀)-C_(L)(t_(N)), where N is any integer, areunfolded into an example array 614 of example columns 616A-616N andexample rows 618A-618N. Each column (e.g., 616A) and row (e.g., 618A) inthe array 614 is represented by an example block 620 corresponding toimage data (e.g., temperature, etc.) for a portion of the device 110,such as the selected volume 405, at a particular time.

The first input data vector 610A represents image data for a selectedvolume 405 at a first time (t₀). Each of the blocks 620 of the selectedvolume 405 has a uniform first temperature, expressed as uniform fill inthe first input data vector 610A of FIG. 6. The second input data vector610B represents image data for a selected volume 405 at a second time(t₁). Each block 620 of the selected volume 405, save block 622, has afirst temperature, expressed as uniform fill in the second input datavector 610B of FIG. 6. Block 622 is shown to have a second temperaturehigher than the first temperature, expressed as different fill than thatof the first temperature. The third input data vector 610C representsimage data for a selected volume 405 at a third time (t₂). Each block620 of the selected volume 405, save blocks 624 and 626, has a firsttemperature, expressed as uniform fill in the third input data vector610C of FIG. 6. Block 624 is shown to have a third temperature higherthan the second temperature, expressed as different fill than that ofthe second temperature. Block 626 is shown to have a second temperaturehigher than the first temperature, expressed as different fill than thatof the first temperature.

The fourth input data vector 610D represents image data for a selectedvolume 405 at a fourth time (t₃). Some blocks 620 of the selected volume405 have a first temperature, block 628 is shown to have a fourthtemperature higher than the third temperature, and block 630 is shown tohave the third temperature, with each of the different temperaturesbeing expressed using a different fill. The fifth input data vector 610Erepresents image data for a selected volume 405 at a fifth time (t₄).Some blocks 620 of the selected volume 405 are shown to have the firsttemperature, block 632 is shown to have the fourth temperature, andblock 634 is shown to have the third temperature higher, with each ofthe different temperatures being expressed using a different fill. Block636 of the fifth input data vector 610E is shown to have the secondtemperature. The sixth input data vector 610F represents image data fora selected volume 405 at a sixth time (t₅). Some blocks 620 of theselected volume 405 are shown to have the first temperature, block 638is shown to have the fourth temperature, and block 640 and block 642 areshown to have the third temperature.

FIG. 7 illustrates an example training model 700 used to form and/orupdate an impression 254 for device 110 from the image data from theimage sensor 120 of FIGS. 1 and/or 3A-3C by the governor 150 of FIGS.1-2.

FIG. 7 shows example input data vectors (C_(L)(t_(X))) over time for thedevice 110, with an example first training sample 702 occurring at afirst time, an example second training sample 704 occurring at a secondtime and an example third training sample 706 occurring at a third time.In some examples, a plurality of training samples are used to developthe impression 254 to account for normal variations, such as variabilityin sensor measurements (e.g., sensor accuracy, etc.) and/ornon-substantive variations in operational system performance.

In each of the first training sample 702, the second training sample 704and the third training sample 704, the rows 618F, 618G and 618J includeblocks 620 exhibiting higher temperatures than the remaining blocks 620,which are all at the first temperature 708. In the first training sample702, row 618F shows that the block 620 at column 616A (C_(L)(t-3)) has asecond temperature 710 higher than the first temperature 708, the block620 at column 616B (C_(L)(t-2)) has a third temperature 712 higher thanthe second temperature 710, the block 620 at column 616C (C_(L)(t-1))has a fourth temperature 714 higher than the third temperature 712 andthe block 620 at column 616N (C_(L)(t)) is shown to have the fourthtemperature 714. Row 618G of the first training sample 702 shows thatthe block 620 at column 616A (C_(L)(t-3)) has the first temperature 708,the block 620 at column 616B (C_(L)(t-2)) has the second temperature710, the block 620 at column 616C (C_(L)(t-1)) has the third temperature712 and the block 620 at column 616N (C_(L)(t)) is shown to have thethird temperature 712. Row 618J of the first training sample 702 showsthat the block 620 at column 616N (C_(L)(t)) has the second temperature710.

In the second training sample 704 and the third training sample 706, row618F shows that the block 620 at column 616A (C_(L)(t-3)) has thirdtemperature 712, the block 620 at column 616B (C_(L)(t-2)) has thefourth temperature, the block 620 at column 616C (C_(L)(t-1)) has thefourth temperature and the block 620 at column 616N (C_(L)(t)) has thefourth temperature 714. Row 618G of the first training sample 702 showsthat the block 620 at column 616A (C_(L)(t-3)) has the secondtemperature, the block 620 at column 616B (C_(L)(t-2)) has the thirdtemperature, the block 620 at column 616C (C_(L)(t-1)) has the thirdtemperature and the block 620 at column 616N (C_(L)(t)) has the thirdtemperature. Row 618J of the second training sample 704 shows that theblock 620 at column 616C (C_(L)(t-1)) has the second temperature and theblock 620 at column 616N (C_(L)(t)) has the third temperature.

In some examples, such as is shown in the example of FIG. 7, image datafrom the image sensor 120, or derivatives thereof, is communicated fromthe device 110, via the governor 150 and the communication device 155,to the external machines 170 (e.g., computing devices of any kind,computer, server, etc.) via the RF broadcast tower 160 and/or network165.

As noted above, in some examples, the training of the impression 254includes implementation of neural networks, decision trees, supportvector machines (SVMs), and/or other machine learning application. Thetraining of the impression 254 may include, for example, updating of amatrix or coefficients responsive to image data from the image sensor120 to modify the impression 254 to accommodate valid changes in imagedata. In the example of FIG. 7, a training model (e.g., back propagationmodel, decision tree based model, etc.) is applied to historic imagedata (e.g., the first training sample 702 and the second training sample704) and current image data (e.g., the third training sample 704) sothat the impression or trained model would, in view of the historicimage data, predict image data close to the current image data. Stateddifferently, if it is desired to train the impression 254 to predictfuture values, the training process is advantageously informed via aplurality of examples of result (x1, y1), (x2, y2), . . . (xN, yN),where x would be a vector of input data, and y would be a desirableoutput. During the training process of the impression 254, impression254 coefficients are adjusted to incorporate “knowledge” of input imagedata patterns until the impression starts to approximate, or in factapproximates, a desirable output. At that point, further input of imagedata into the impression 254 should yield a resulting output close toactual image data in the case of normal, known behavior of the device110, whereas a resulting output diverging from the actual image data(e.g., incorrect predictions beyond a predetermined percentagedifference from the actual image data, incorrect predictions beyond apredetermined threshold from the actual image data, etc.) can beidentified.

In some examples, the training of the impression 254 is during normaloperation of the device 110 for a sufficient time for the impression 254to learn actual behavior of the device 110 in advance of any potentialfor anomalous behavior of the device 110. In some examples, the device110 may include a vendor-supplied impression 254 (e.g., values ofcoefficients for a pre-trained model, etc.) in the memory 250.

FIG. 8 illustrates another representation of implementation of the imagedata from the image sensor 120 of FIGS. 1 and 3A-3C by the governor 150of FIGS. 1-2 in accord with the example instructions of FIG. 5A and/orFIG. 5B. In particular, FIG. 8 represents a monitoring phase of animpression 254 during operation of device 110. On the left side of FIG.8 is a representation of an example history 805 for the device 110(e.g., an example autonomous or manned device, etc.). In FIG. 8, theexample history 805 corresponds to training sample 704 described abovein relation to FIG. 7. In some examples, the input data vectors(C_(L)(t_(X))) (e.g., C_(L)(t-3), C_(L)(t-2) and C_(L)(t-1) of FIG. 8)are analyzed via a function F (e.g., a predictive model derived from thetraining samples, a predictive function in the form of a neural network,an adjustment via a decision tree learning model, an association rulelearning model, etc.) to yield an impression 254 or trained modelincluding an example array 830 of expected values 831A-831P (e.g., datavectors N*_(L)(t) containing a predicted distribution based on historicvalues, etc.). N*_(L)(t) and (C_(L)(t_(X))) are normalized to a commonbasis.

The example array 850 of FIG. 8, the image data corresponds to actualtemperatures and, more particularly, example rows 851A-851E indicate thefirst temperature 708, the example row 851F indicates the fourthtemperature 714, the example row 851G indicates the fourth temperature714, the example rows 851H-851I indicate the first temperature 708, theexample row 851J indicate the fourth temperature 714 and the examplerows 851K-851P indicate the first temperature 708.

The expected values 831A-831P are then compared by the impressioncomparator 230 to the example rows 851A-851P of data vectors (C_(L)(t))in example array 850, corresponding to actual image data.

In some examples, the data vectors (C_(L)(t)) of rows 851A-851P of array850 (e.g., actual values of image data) are compared to the data vectors(N*_(L)(t)) of the rows of expected values 831A-831P of array 830 (e.g.,the expected values of impression 254) to determine if any comparisonbetween corresponding data vectors (e.g., a comparison of the datavector of row 851A to the data vector of row 831A, etc.) is greater thana threshold difference. In some examples, where the data vectorsrepresent temperatures (e.g., absolute temperatures, temperaturegradients, etc.), the threshold difference may be expressed in adifference in temperatures (e.g., a difference of 1° F., 2° F., 3° F. .. . 10° F., 20° F., 30° F., etc.). To illustrate, the data vector C₁₀(t)of row 851J is shown to correspond to the fourth temperature 714,whereas the data vector N*₁₀(t) of row 831J (e.g., the impression) isshown to correspond to the third temperature 712, indicating adifference therebetween. In some examples, where the data vectorsrepresent distances or dimensions, the threshold difference may beexpressed in a difference in dimension (e.g., a difference of 0.1 mm,0.2 mm, 0.3 mm, etc.). In some examples, the threshold difference isdetermined via a sum of absolute differences, a sum of differencessquared, or the like.

FIG. 9 is a block diagram of an example processor platform 900 capableof executing the instructions of FIGS. 5A-5B to implement the examplegovernor 150 of FIG. 2. The processor platform 900 can be, for example,a server, a personal computer, a mobile device (e.g., a cell phone, asmart phone, a tablet such as an iPad™), a vehicle controller, a dronecontroller, a robotic device controller, a personal digital assistant(PDA), an Internet appliance, or any other type of computing device.

The processor platform 900 of the illustrated example includes aprocessor 912. The processor 912 of the illustrated example is hardware.For example, the processor 912 can be implemented by one or moreintegrated circuits, logic circuits, microprocessors or controllers fromany desired family or manufacturer. The hardware processor may be asemiconductor based (e.g., silicon based) device. In this example, theprocessor 912 implements the governor 150, the image manager 210, theimpression manager 220, and the impression comparator 230.

The processor 912 of the illustrated example includes a local memory 913(e.g., a cache). The processor 912 of the illustrated example is incommunication with a main memory including a volatile memory 914 and anon-volatile memory 916 via a bus 918. The volatile memory 914 may beimplemented by Synchronous Dynamic Random Access Memory (SDRAM), DynamicRandom Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM)and/or any other type of random access memory device. The non-volatilememory 916 may be implemented by flash memory and/or any other desiredtype of memory device. Access to the volatile memory and non-volatilememory 914, 916, local memory and/or main memory is controlled by amemory controller.

The processor platform 900 of the illustrated example also includes aninterface circuit 920. The interface circuit 920 may be implemented byany type of interface standard, such as an Ethernet interface, auniversal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices 922 are connectedto the interface circuit 920. The input device(s) 922 permit(s) a userto enter data and/or commands into the processor 912. The inputdevice(s) can be implemented by, for example, an audio sensor, amicrophone, a camera (still or video), a keyboard, a button, a mouse, atouchscreen, a track-pad, a trackball, isopoint and/or a voicerecognition system.

One or more output devices 924 are also connected to the interfacecircuit 920 of the illustrated example. The output devices 924 can beimplemented, for example, by display devices (e.g., a light emittingdiode (LED), an organic light emitting diode (OLED), a liquid crystaldisplay, a touchscreen, a tactile output device, a printer and/orspeakers). The interface circuit 920 of the illustrated example, thus,typically includes a graphics driver card, a graphics driver chip and/ora graphics driver processor.

The interface circuit 920 of the illustrated example also includes acommunication device such as a transmitter, a receiver, a transceiver, amodem and/or network interface card to facilitate exchange of data withexternal machines (e.g., computing devices of any kind) via a network165 (e.g., an Ethernet connection, a digital subscriber line (DSL), atelephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform 900 of the illustrated example also includes oneor more mass storage devices 928 for storing software and/or data.Examples of such mass storage devices 928 include floppy disk drives,hard drive disks, compact disk drives, Blu-ray disk drives, RAIDsystems, and digital versatile disk (DVD) drives.

The coded instructions 932 of FIGS. 5A-5B may be stored in the massstorage device 928, in the volatile memory 914, in the non-volatilememory 916, and/or on a removable tangible computer readable storagemedium such as a CD or DVD.

From the foregoing, it will be appreciated that example methods,apparatus and articles of manufacture have been disclosed that enableearly detection of a wide range of known and/or unforeseeablemalfunctions and that enable early intervention. This is particularlyadvantageous for autonomous devices, as it can help to detect problemsbefore the autonomous devices become unrecoverable, and devices having ahigh degree of similitude, as knowledge gained on one autonomous and/ormanned device can be applied to other similar autonomous and/or manneddevices.

Example 1 is a monitoring system including an image sensor to obtainimage data of a device and a governor to cause the image sensor toobtain image data of the device, to form an impression from the imagedata, to use the impression and the image data to generate expectedvalues for a state of the device at a predetermined time, and to compareimage data for the structure at the predetermined time with thegenerated expected values.

Example 2 includes the monitoring system as defined in example 1,wherein the image sensor includes at least one of a thermal imagesensor, a spatial image sensor, and an optical image sensor and whereinthe image data includes at least one of thermal image data, spatialimage data, and optical image data.

Example 3 includes the monitoring system as defined in example 1 orexample 2, wherein the image sensor includes a thermal image sensor anda spatial image sensor, wherein the image data includes thermal imagedata and spatial image data.

Example 4 includes the monitoring system as defined in any of examples1-3, wherein the image sensor is to obtain the image data of the device,via the image sensor, during operation of the device.

Example 5 includes the monitoring system as defined in any of examples1-4, wherein the image sensor is to obtain the image data of the device,via the image sensor, between operation cycles of the device.

Example 6 includes the monitoring system as defined in any of examples1-5, wherein the impression is formed from a plurality of sets of theimage data.

Example 7 includes the monitoring system as defined in any of examples1-6, further including a non-transitory machine readable medium to storeat least one of the impression or the image data.

Example 8 includes the monitoring system as defined in any of examples1-7, wherein the impression includes a mapping of at least one ofspatial gradients or thermal gradients within a selected volume of astructure of the device onto coefficients vectors to permit comparison,over respective basis spaces, to actual values from the image data.

Example 9 includes the monitoring system as defined in any of examples1-8, wherein the governor is to, following the comparison of the imagedata for the device at the predetermined time with the generatedexpected values from the impression, determine whether telemetry datafrom instruments monitoring one or more systems or subsystems of thedevice is within acceptable limits.

Example 10 includes the monitoring system as defined in any of examples1-9, further including a communication device to communicate, to aremote device, at least one of the impression, the image data, or adeviation report relating to the impression.

Example 11 includes the monitoring system as defined in any of examples1-10, wherein the governor is to generate a deviation report responsiveto a difference between an expected value generated from by theimpression and an actual value of the image data.

Example 12 includes the monitoring system as defined in any of examples1-11, wherein the remote device includes another device similarlyconfigured to the device.

Example 13 includes the monitoring system as defined in any of examples1-12, wherein the remote device includes a central server or service incommunication with a plurality of similarly configured device.

Example 14 is a method for automated monitoring of a device, includingimaging a device during an operating state of the device to obtain firstimage data, forming an impression from the first image data, imaging thedevice during a subsequent operating state of the device to obtainsecond image data, estimating values for the second image data using theimpression, and comparing the estimated values for the second image datato actual values for the second image data.

Example 15 includes the method for automated monitoring of claim 14, andfurther includes determining if a difference between the estimatedvalues for the second image data and the actual values for the secondimage data is less than a threshold difference.

Example 16 includes the method for automated monitoring of claim 14 orclaim 15, and further includes determining if telemetry data from asensor of the device is within acceptable limits.

Example 17 includes the method for automated monitoring of any of claims14-16, and further includes updating the impression to incorporate thesecond image data as good data if the telemetry data is withinacceptable limits.

Example 18 includes the method for automated monitoring of any of claims14-17, and further includes updating the impression to incorporate thesecond image data as bad data if telemetry data from a sensor of thedevice is not within acceptable limits.

Example 19 includes the method for automated monitoring of any of claims14-18, and further includes outputting a deviation report.

Example 20 includes the method for automated monitoring of any of claims14-19, and further includes, following a determination that thedifference between the estimated values for the second image data andthe actual values for the second image data are less than the thresholddifference, comparing the estimated values for the second image data toknown values corresponding to a bad outcome to determine if theestimated values for the second image data correspond to the badoutcome.

Example 21 includes the method for automated monitoring of any of claims14-20, and further includes outputting a deviation report if theestimated values for the second image data are determined to correspondto the bad outcome.

Example 22 includes the method for automated monitoring of any of claims14-21, and further includes determining if telemetry data from a sensorof the device is within acceptable limits if the estimated values forthe second image data are determined not to correspond to the badoutcome.

Example 23 includes the method for automated monitoring of any of claims14-22, and further includes updating the impression to incorporate thesecond image data as good data if the telemetry data is withinacceptable limits.

Example 24 is a system including an imaging means to obtain image dataof a device and a governing means to cause the imaging means to obtainimage data of the device, to form an impression from the image data, touse the impression and the image data to generate expected values for astate of the device at a predetermined time, and to compare image datafor the device at the predetermined time with the generated expectedvalues.

Example 25 includes the system of claim 24, wherein the imaging meansincludes at least one of a thermal imaging means, a spatial imagingmeans, and an optical imaging means and wherein the image data includesat least one of thermal image data, spatial image data, and opticalimage data.

Example 26 includes the system of claim 24 or claim 25, wherein theimaging means includes a thermal imaging means and a spatial imagingmeans, and wherein the image data includes thermal image data andspatial image data.

Example 27 includes the system of claims 24-26, wherein the imagingmeans is to obtain the image data of the device, via the imaging means,during operation of the device.

Example 28 includes the system of claims 24-27, wherein the imagingmeans is to obtain the image data of the device, via the imaging means,between operation cycles of the device.

Example 29 includes the system of claims 24-28, wherein the impressionis formed from a plurality of sets of the image data.

Example 30 includes the system of claims 24-29, further including acommunication means to communicate to a remote device at least one ofthe impression, the image data, or a deviation report relating to theimpression in response to a difference between expected values generatedfrom the impression and actual values of the image data exceeding athreshold difference.

Example 31 is a non-transitory machine readable medium comprisingexecutable instructions that, when executed, cause at least oneprocessor to image a device during an operating state of the device toobtain first image data, form an impression from the first image data,image the device during the operating state of the device to obtainsecond image data, estimate values for the second image data using theimpression and compare the estimated values for the second image data toactual values for the second image data.

Example 32 includes the non-transitory machine readable medium of claim31, and further includes executable instructions that, when executed,cause at least one processor to determine if a difference between theestimated values for the second image and the actual values for thesecond image data is less than a threshold difference.

Example 33 includes the non-transitory machine readable medium of claim31 or claim 32, and further includes executable instructions that, whenexecuted, cause at least one processor to determine if telemetry datafrom a sensor of the device is within acceptable limits.

Example 34 includes the non-transitory machine readable medium of any ofclaims 31-33, and further includes executable instructions that, whenexecuted, cause at least one processor to update the impression toincorporate the image data as good data if the telemetry data is withinacceptable limits.

Example 35 includes the non-transitory machine readable medium of any ofclaims 31-34, and further includes executable instructions that, whenexecuted, cause at least one processor to update the impression toincorporate the image data as bad data if telemetry data from a sensorof the device is not within acceptable limits.

Example 36 includes the non-transitory machine readable medium of any ofclaims 31-35, and further includes executable instructions that, whenexecuted, cause at least one processor to output a deviation report if adifference between the estimated values for the second image and theactual values for the second image data is greater than a thresholddifference.

Example 37 is a monitoring system including an image sensor to obtainimage data of a device and a governor to cause the image sensor toobtain image data of the device, to form an impression from the imagedata, to use the impression and the image data to determine a verdict.

Example 38 includes the monitoring system of claim 37, wherein theverdict is determined by using the impression and the image data togenerate expected values for a state of the device at a predeterminedtime and to compare image data for the device at the predetermined timewith the generated expected values.

Example 39 includes the monitoring system of claim 37 or claim 38,wherein the impression is to determine the verdict using the image datadirectly.

Although certain example methods, device and articles of manufacturehave been disclosed herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods, deviceand articles of manufacture fairly falling within the scope of theclaims of this patent. For instance, while examples herein havedisclosed implementation of an image sensor 120 to obtain image datarepresentative of a temperature of one or more objects in a specifiedvolume 405, in some examples contactless thermometers (pyrometers)and/or contact thermometer arrays (a plurality of thermometers) may beplaced on surfaces of one or more parts to provide temperature data inlieu of image data.

What is claimed is:
 1. A monitoring system, comprising: an image sensorto obtain image data of a device; and a governor to cause the imagesensor to obtain image data of the device, to form an impression fromthe image data, to use the impression and the image data to generateexpected values for a state of the device at a predetermined time, andto compare image data for the device at the predetermined time with thegenerated expected values.
 2. The monitoring system of claim 1, whereinthe image sensor includes at least one of a thermal image sensor, aspatial image sensor, and an optical image sensor and wherein the imagedata includes at least one of thermal image data, spatial image data,and optical image data.
 3. The monitoring system of claim 2, wherein theimage sensor includes a thermal image sensor and a spatial image sensor,wherein the image data includes thermal image data and spatial imagedata.
 4. The monitoring system of claim 2, wherein the image sensor isto obtain the image data of the device, via the image sensor, duringoperation of the device.
 5. The monitoring system of claim 2, whereinthe image sensor is to obtain the image data of the device, via theimage sensor, between operation cycles of the device.
 6. The monitoringsystem of claim 2, wherein the impression is formed from a plurality ofsets of the image data.
 7. The monitoring system of claim 2, furtherincluding a non-transitory machine readable medium to store at least oneof the impression or the image data.
 8. The monitoring system of claim2, wherein the impression includes a mapping of at least one of spatialgradients or thermal gradients within a selected volume of a structureof the device onto coefficients vectors to permit comparison, overrespective basis spaces, to actual values from the image data.
 9. Themonitoring system of claim 2, further including a communication deviceto communicate, to a remote device, at least one of the impression, theimage data, or a deviation report relating to the impression.
 10. Themonitoring system of claim 9, wherein the governor is to generate adeviation report responsive to a difference between expected valuesgenerated from the impression and actual values of the image data.
 11. Amethod for automated monitoring of a device, comprising: imaging adevice during an operating state of the device to obtain first imagedata; forming an impression from the first image data; imaging thedevice during a subsequent operating state of the device to obtainsecond image data; estimating values for the second image data using theimpression; and comparing the estimated values for the second image datato actual values for the second image data.
 12. The method of claim 11,further including determining if a difference between the estimatedvalues for the second image data and the actual values for the secondimage data is less than a threshold difference.
 13. The method of claim12, further including determining if telemetry data from a sensor of thedevice is within acceptable limits.
 14. The method of claim 13, furtherincluding updating the impression to incorporate the second image dataas good data if the telemetry data is within acceptable limits.
 15. Themethod of claim 13, further including updating the impression toincorporate the second image data as bad data if telemetry data from asensor of the device is not within acceptable limits.
 16. The method ofclaim 15, further including outputting a deviation report.
 17. Themethod of claim 12, further including, following a determination thatthe difference between the estimated values for the second image dataand the actual values for the second image data are less than thethreshold difference, comparing the estimated values for the secondimage data to known values corresponding to a bad outcome to determineif the estimated values for the second image data correspond to the badoutcome.
 18. A monitoring system, comprising: an image sensor to obtainimage data of a device; and a governor to cause the image sensor toobtain image data of the device, to form an impression from the imagedata, to use the impression and the image data to determine a verdict.19. The monitoring system of claim 18, wherein the verdict is determinedby using the impression and the image data to generate expected valuesfor a state of the device at a predetermined time and to compare imagedata for the device at the predetermined time with the generatedexpected values.
 20. The monitoring system of claim 18, wherein theimpression is to determine the verdict using the image data directly.